Log Analysis and Visulization

In [1]:
import utils
import os
from os import path
import datetime
import pandas as pd

Result

  • Statistics Summary
    • transaction per second
    • time per block
    • size
    • gas
  • Features vs Time / Block Height (with all shards / per shard)
    • staking transaction per second
    • total transaction per second
    • transaction per second
    • time per block
    • size
    • gas
In [2]:
if __name__ == "__main__":
    
    date = datetime.date.today()-datetime.timedelta(days=1)
    name = "ostn_"+date.strftime("%Y_%m_%d")
    fig_dir = "../../graphs/test_logs/ostn/" + date.strftime("%Y_%m_%d") + "/"
    log_dir = "/home/ubuntu/jupyter/monitor/mini_explorer/data/" + name
    html_dir = "../../docs/graphs/test_logs/ostn/" + date.strftime("%Y_%m_%d") + "/"

    data = [html_dir, fig_dir]
    for directory in data:
        if not path.exists(directory):
            try:
                os.makedirs(directory)
            except:
                print("Could not make directory")
                exit(1)

    data = utils.read_data(log_dir)
    shard = utils.data_processing(data)
    
    utils.set_config({
        'ignore_printing_statistics_summary': False,
        'ignore_drawing_features_vs_time_per_shard' : True,
        'ignore_drawing_features_vs_block_per_shard' : True,
        'ignore_drawing_staking_features_vs_time_per_shard' : True,
        'ignore_drawing_staking_features_vs_block_per_shard' : True,
        'ignore_drawing_features_vs_time' : False,
        'ignore_drawing_features_vs_block' : True
    })
    
    config = utils.config
    ## get the index of the new chain started
    index = []
    for s in shard:
        i = s[s['time_per_block'] < 0].index.tolist()   
        if len(i) == 0:
            break
        i.insert(0,0)
        i.append(len(s))
        index.append(i)
    
    if len(index) == 0:
        new = pd.concat(shard)
        new.reset_index(inplace = True, drop = True)
        print("transaction for ", date, new['transactions'].sum())
        utils.visualization(new, fig_dir, html_dir, config)
    else:    
        for i in range(len(index[0])-1):
            df = []
            for s in range(len(shard)):
                df.append(shard[s].iloc[index[s][i]+1:index[s][i+1]])
            new = pd.concat(df)

            fig_path = fig_dir + "chain_" + str(i) + "/"
            if not os.path.exists(fig_path):
                os.makedirs(fig_path)

            utils.visualization(new, fig_path, html_dir, config)
Statistics summary for shard 0
==================================
Total data points: 292
             size  gas  transaction_per_second  time_per_block
mean   755.708904  0.0                     0.0        1.634344
std    727.017090  0.0                     0.0        0.810100
min    677.000000  0.0                     0.0       -0.104892
25%    677.000000  0.0                     0.0        1.333333
50%    677.000000  0.0                     0.0        1.500000
75%    677.000000  0.0                     0.0        1.666667
max   9474.000000  0.0                     0.0        6.500000

Statistics summary for shard 1
==================================
Total data points: 375
            size  gas  transaction_per_second  time_per_block
mean  677.277333  0.0                     0.0        4.355384
std     2.674477  0.0                     0.0       40.575918
min   677.000000  0.0                     0.0       -0.036839
25%   677.000000  0.0                     0.0        0.666667
50%   677.000000  0.0                     0.0        0.714286
75%   677.000000  0.0                     0.0        3.000000
max   703.000000  0.0                     0.0      758.000000

Statistics summary for shard 2
==================================
Total data points: 1
       size  gas  transaction_per_second  time_per_block
mean  703.0  0.0                     0.0       -0.382607
std     NaN  NaN                     NaN             NaN
min   703.0  0.0                     0.0       -0.382607
25%   703.0  0.0                     0.0       -0.382607
50%   703.0  0.0                     0.0       -0.382607
75%   703.0  0.0                     0.0       -0.382607
max   703.0  0.0                     0.0       -0.382607

Statistics summary for shard 3
==================================
Total data points: 122
             size  gas  transaction_per_second  time_per_block
mean   821.393443  0.0                     0.0       21.040810
std   1121.654869  0.0                     0.0       75.681269
min    677.000000  0.0                     0.0       -0.021151
25%    677.000000  0.0                     0.0       12.000000
50%    677.000000  0.0                     0.0       13.000000
75%    677.000000  0.0                     0.0       13.000000
max   9474.000000  0.0                     0.0      667.000000

Features vs Time
PNG saved in ../../graphs/test_logs/ostn/2020_05_16/chain_0/
==================================
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
Statistics summary for shard 0
==================================
Total data points: 1369
             size  gas  transaction_per_second  time_per_block
mean   889.771366  0.0                     0.0        3.997686
std    459.707744  0.0                     0.0        0.115858
min    677.000000  0.0                     0.0       -0.167742
25%    677.000000  0.0                     0.0        4.000000
50%    707.000000  0.0                     0.0        4.000000
75%   1120.000000  0.0                     0.0        4.000000
max   9917.000000  0.0                     0.0        5.000000

Statistics summary for shard 1
==================================
Total data points: 1370
             size  gas  transaction_per_second  time_per_block
mean   698.002920  0.0                     0.0        3.997061
std    335.927146  0.0                     0.0        0.108801
min    677.000000  0.0                     0.0       -0.027097
25%    677.000000  0.0                     0.0        4.000000
50%    677.000000  0.0                     0.0        4.000000
75%    703.000000  0.0                     0.0        4.000000
max   9474.000000  0.0                     0.0        4.000000

Statistics summary for shard 2
==================================
Total data points: 1369
             size  gas  transaction_per_second  time_per_block
mean   704.356465  0.0                     0.0        3.997058
std    409.864612  0.0                     0.0        0.108858
min    677.000000  0.0                     0.0       -0.027760
25%    677.000000  0.0                     0.0        4.000000
50%    677.000000  0.0                     0.0        4.000000
75%    703.000000  0.0                     0.0        4.000000
max   9474.000000  0.0                     0.0        4.000000

Statistics summary for shard 3
==================================
Total data points: 1375
             size  gas  transaction_per_second  time_per_block
mean   697.858182  0.0                     0.0        3.997798
std    333.532037  0.0                     0.0        0.111951
min    677.000000  0.0                     0.0       -0.028296
25%    677.000000  0.0                     0.0        4.000000
50%    677.000000  0.0                     0.0        4.000000
75%    703.000000  0.0                     0.0        4.000000
max   9474.000000  0.0                     0.0        5.000000

Features vs Time
PNG saved in ../../graphs/test_logs/ostn/2020_05_16/chain_1/
==================================
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
Statistics summary for shard 0
==================================
Total data points: 9
            size  gas  transaction_per_second  time_per_block
mean  702.555556  0.0                     0.0       58.972222
std     1.333333  0.0                     0.0      146.551636
min   699.000000  0.0                     0.0       -3.250000
25%   703.000000  0.0                     0.0        4.000000
50%   703.000000  0.0                     0.0        4.000000
75%   703.000000  0.0                     0.0        4.000000
max   703.000000  0.0                     0.0      446.000000

Statistics summary for shard 1
==================================
Total data points: 189
            size  gas  transaction_per_second  time_per_block
mean  703.835979  0.0                     0.0        4.041290
std     0.989077  0.0                     0.0        0.589295
min   703.000000  0.0                     0.0       -0.196262
25%   703.000000  0.0                     0.0        4.000000
50%   703.000000  0.0                     0.0        4.000000
75%   705.000000  0.0                     0.0        4.000000
max   705.000000  0.0                     0.0        8.000000

Statistics summary for shard 2
==================================
Total data points: 190
            size  gas  transaction_per_second  time_per_block
mean  703.831579  0.0                     0.0        3.977816
std     0.988319  0.0                     0.0        0.305785
min   703.000000  0.0                     0.0       -0.214953
25%   703.000000  0.0                     0.0        4.000000
50%   703.000000  0.0                     0.0        4.000000
75%   705.000000  0.0                     0.0        4.000000
max   705.000000  0.0                     0.0        4.000000

Statistics summary for shard 3
==================================
Total data points: 184
            size  gas  transaction_per_second  time_per_block
mean  703.804348  0.0                     0.0        3.976981
std     0.983349  0.0                     0.0        0.312251
min   703.000000  0.0                     0.0       -0.235577
25%   703.000000  0.0                     0.0        4.000000
50%   703.000000  0.0                     0.0        4.000000
75%   705.000000  0.0                     0.0        4.000000
max   705.000000  0.0                     0.0        4.000000

Features vs Time
PNG saved in ../../graphs/test_logs/ostn/2020_05_16/chain_2/
==================================
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
Statistics summary for shard 0
==================================
Total data points: 3
       size  gas  transaction_per_second  time_per_block
mean  703.0  0.0                     0.0      -23.000000
std     0.0  0.0                     0.0       54.561891
min   703.0  0.0                     0.0      -86.000000
25%   703.0  0.0                     0.0      -39.000000
50%   703.0  0.0                     0.0        8.000000
75%   703.0  0.0                     0.0        8.500000
max   703.0  0.0                     0.0        9.000000

Statistics summary for shard 1
==================================
Total data points: 36
       size  gas  transaction_per_second  time_per_block
mean  703.0  0.0                     0.0        7.739198
std     0.0  0.0                     0.0        1.564815
min   703.0  0.0                     0.0       -1.388889
25%   703.0  0.0                     0.0        8.000000
50%   703.0  0.0                     0.0        8.000000
75%   703.0  0.0                     0.0        8.000000
max   703.0  0.0                     0.0        8.000000

Statistics summary for shard 2
==================================
Total data points: 42
       size  gas  transaction_per_second  time_per_block
mean  703.0  0.0                     0.0        7.784580
std     0.0  0.0                     0.0        1.396078
min   703.0  0.0                     0.0       -1.047619
25%   703.0  0.0                     0.0        8.000000
50%   703.0  0.0                     0.0        8.000000
75%   703.0  0.0                     0.0        8.000000
max   703.0  0.0                     0.0        8.000000

Statistics summary for shard 3
==================================
Total data points: 45
       size  gas  transaction_per_second  time_per_block
mean  703.0  0.0                     0.0        7.797531
std     0.0  0.0                     0.0        1.358204
min   703.0  0.0                     0.0       -1.111111
25%   703.0  0.0                     0.0        8.000000
50%   703.0  0.0                     0.0        8.000000
75%   703.0  0.0                     0.0        8.000000
max   703.0  0.0                     0.0        8.000000

Features vs Time
PNG saved in ../../graphs/test_logs/ostn/2020_05_16/chain_3/
==================================
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
Statistics summary for shard 0
==================================
Total data points: 1
       size  gas  transaction_per_second  time_per_block
mean  703.0  0.0                     0.0         -3165.0
std     NaN  NaN                     NaN             NaN
min   703.0  0.0                     0.0         -3165.0
25%   703.0  0.0                     0.0         -3165.0
50%   703.0  0.0                     0.0         -3165.0
75%   703.0  0.0                     0.0         -3165.0
max   703.0  0.0                     0.0         -3165.0

Statistics summary for shard 1
==================================
Total data points: 393
            size  gas  transaction_per_second  time_per_block
mean  705.076336  0.0                     0.0        7.979346
std     1.642632  0.0                     0.0        0.409451
min   703.000000  0.0                     0.0       -0.117048
25%   703.000000  0.0                     0.0        8.000000
50%   705.000000  0.0                     0.0        8.000000
75%   707.000000  0.0                     0.0        8.000000
max   707.000000  0.0                     0.0        8.000000

Statistics summary for shard 2
==================================
Total data points: 387
            size  gas  transaction_per_second  time_per_block
mean  705.036176  0.0                     0.0        7.979022
std     1.662987  0.0                     0.0        0.412689
min   699.000000  0.0                     0.0       -0.118557
25%   703.000000  0.0                     0.0        8.000000
50%   705.000000  0.0                     0.0        8.000000
75%   707.000000  0.0                     0.0        8.000000
max   707.000000  0.0                     0.0        8.000000

Statistics summary for shard 3
==================================
Total data points: 383
            size  gas  transaction_per_second  time_per_block
mean  705.026110  0.0                     0.0        7.978792
std     1.633852  0.0                     0.0        0.415051
min   703.000000  0.0                     0.0       -0.122715
25%   703.000000  0.0                     0.0        8.000000
50%   705.000000  0.0                     0.0        8.000000
75%   707.000000  0.0                     0.0        8.000000
max   707.000000  0.0                     0.0        8.000000

Features vs Time
PNG saved in ../../graphs/test_logs/ostn/2020_05_16/chain_4/
==================================
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
Statistics summary for shard 0
==================================
Total data points: 1
       size  gas  transaction_per_second  time_per_block
mean  699.0  0.0                     0.0          -203.5
std     NaN  NaN                     NaN             NaN
min   699.0  0.0                     0.0          -203.5
25%   699.0  0.0                     0.0          -203.5
50%   699.0  0.0                     0.0          -203.5
75%   699.0  0.0                     0.0          -203.5
max   699.0  0.0                     0.0          -203.5

Statistics summary for shard 1
==================================
Total data points: 4724
             size  gas  transaction_per_second  time_per_block
mean   681.514606  0.0                     0.0        7.998304
std    126.855884  0.0                     0.0        0.116549
min    677.000000  0.0                     0.0       -0.010584
25%    677.000000  0.0                     0.0        8.000000
50%    677.000000  0.0                     0.0        8.000000
75%    677.000000  0.0                     0.0        8.000000
max   9380.000000  0.0                     0.0        8.000000

Statistics summary for shard 2
==================================
Total data points: 4726
             size  gas  transaction_per_second  time_per_block
mean   681.518197  0.0                     0.0        7.998305
std    126.829436  0.0                     0.0        0.116531
min    677.000000  0.0                     0.0       -0.011005
25%    677.000000  0.0                     0.0        8.000000
50%    677.000000  0.0                     0.0        8.000000
75%    677.000000  0.0                     0.0        8.000000
max   9380.000000  0.0                     0.0        8.000000

Statistics summary for shard 3
==================================
Total data points: 4726
             size  gas  transaction_per_second  time_per_block
mean   681.512696  0.0                     0.0        7.998305
std    126.829068  0.0                     0.0        0.116503
min    677.000000  0.0                     0.0       -0.009099
25%    677.000000  0.0                     0.0        8.000000
50%    677.000000  0.0                     0.0        8.000000
75%    677.000000  0.0                     0.0        8.000000
max   9380.000000  0.0                     0.0        8.000000

Features vs Time
PNG saved in ../../graphs/test_logs/ostn/2020_05_16/chain_5/
==================================
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
Statistics summary for shard 0
==================================
Total data points: 373
             size  gas  transaction_per_second  time_per_block
mean   715.324397  0.0                     0.0        8.298332
std    454.945837  0.0                     0.0        6.635043
min    677.000000  0.0                     0.0      -33.489978
25%    677.000000  0.0                     0.0        8.000000
50%    703.000000  0.0                     0.0        8.000000
75%    705.000000  0.0                     0.0        8.000000
max   9474.000000  0.0                     0.0      129.000000

Statistics summary for shard 1
==================================
Total data points: 4121
             size  gas  transaction_per_second  time_per_block
mean   697.116962  0.0                     0.0        8.007876
std    386.716102  0.0                     0.0        0.579734
min    677.000000  0.0                     0.0       -0.043900
25%    677.000000  0.0                     0.0        8.000000
50%    677.000000  0.0                     0.0        8.000000
75%    677.000000  0.0                     0.0        8.000000
max   9474.000000  0.0                     0.0       41.000000

Statistics summary for shard 2
==================================
Total data points: 4121
             size  gas  transaction_per_second  time_per_block
mean   697.117933  0.0                     0.0        8.007388
std    386.716112  0.0                     0.0        0.560622
min    677.000000  0.0                     0.0       -0.054868
25%    677.000000  0.0                     0.0        8.000000
50%    677.000000  0.0                     0.0        8.000000
75%    677.000000  0.0                     0.0        8.000000
max   9474.000000  0.0                     0.0       40.000000

Statistics summary for shard 3
==================================
Total data points: 4120
             size  gas  transaction_per_second  time_per_block
mean   697.121845  0.0                     0.0        8.031793
std    386.762915  0.0                     0.0        1.054594
min    677.000000  0.0                     0.0       -0.011642
25%    677.000000  0.0                     0.0        8.000000
50%    677.000000  0.0                     0.0        8.000000
75%    677.000000  0.0                     0.0        8.000000
max   9474.000000  0.0                     0.0       42.000000

Features vs Time
PNG saved in ../../graphs/test_logs/ostn/2020_05_16/chain_6/
==================================
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
Statistics summary for shard 0
==================================
Total data points: 3839
             size           gas  transaction_per_second  time_per_block
mean  1044.912477     32.821047                0.000195        8.076452
std    454.870542    829.665036                0.004938        4.736953
min    677.000000      0.000000                0.000000        8.000000
25%   1120.000000      0.000000                0.000000        8.000000
50%   1120.000000      0.000000                0.000000        8.000000
75%   1120.000000      0.000000                0.000000        8.000000
max   9917.000000  21000.000000                0.125000      301.500000

Statistics summary for shard 1
==================================
Total data points: 148
            size  gas  transaction_per_second  time_per_block
mean  703.351351  0.0                     0.0        8.087838
std     0.763672  0.0                     0.0        0.284020
min   703.000000  0.0                     0.0        8.000000
25%   703.000000  0.0                     0.0        8.000000
50%   703.000000  0.0                     0.0        8.000000
75%   703.000000  0.0                     0.0        8.000000
max   705.000000  0.0                     0.0        9.000000

Statistics summary for shard 2
==================================
Total data points: 141
            size  gas  transaction_per_second  time_per_block
mean  703.368794  0.0                     0.0             8.0
std     0.778381  0.0                     0.0             0.0
min   703.000000  0.0                     0.0             8.0
25%   703.000000  0.0                     0.0             8.0
50%   703.000000  0.0                     0.0             8.0
75%   703.000000  0.0                     0.0             8.0
max   705.000000  0.0                     0.0             8.0

Statistics summary for shard 3
==================================
Total data points: 148
            size  gas  transaction_per_second  time_per_block
mean  703.324324  0.0                     0.0        8.010135
std     0.739702  0.0                     0.0        0.091651
min   703.000000  0.0                     0.0        8.000000
25%   703.000000  0.0                     0.0        8.000000
50%   703.000000  0.0                     0.0        8.000000
75%   703.000000  0.0                     0.0        8.000000
max   705.000000  0.0                     0.0        9.000000

Features vs Time
PNG saved in ../../graphs/test_logs/ostn/2020_05_16/chain_7/
==================================
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in 
HTML saved in